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ABSTRACT 


Let  A'  =  {A'(I)  :  <  >  0}  be  an  irreducible  semi-Markov  process  (SMP)  on  countable  state  space  E.  For 
fixed  ;  6  E.  let  T(:]  =  inf{f  >  0  :  A-(f-)  z,A'(f)  =  r}  and  set  }'(/)  =  f[X{i)]dt,  where  /:  £  -  R 
is  an  arbitrar>'  function.  Our  objective  is  to  study  the  mixed  moments  of  the  form  ^n:=,r(/.  ),  when 

/,:£  —  /?  is  an  arbitrar}'  function,  for  i  =  1.2 . r,  and  r  is  a  positive  integer.  This  quantity  is  especially 

relevant  to  the  regenerative  simulation.  .\lso.  several  useful  variations  and  generalizations  are  introduced 
and  5fj-ii"d. 
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Recursive  Moment  Formulas  for  Seml*Markov  Processes 

by 

Chia-Hon  Chien 

Department  of  Operations  Research 
Stanford  University 


1.  Introduction 

Let  X  =  {X{1) ;  /  >  0}  be  an  irreducible  semi-Markov  process  (SMP)  on  countable  state  space  E.  For  fixed 
z  6  E,  let  T(z)  =  inf  {/  >  0  ;  A:(<-)  /  z,X(t}  =  z)  and  set 

/•r(x) 

¥{/)=  /(X(t))dl, 

Jo 

where  /:£-•/?  is  an  arbitrary  function.  (Hereafter,  we  will  suppress  the  z  in  T(z)  when  no  confusion  is 
likely.)  Our  objective  is  to  study  the  mixed  moments  of  the  form 

•s| 

when  /,:£-*/?  is  ao  arbitrary  function,  for  •  =  l,2,...,r,  and  r  is  a  positive  integer. 

Hordijk,  Iglehart,  and  Schassberger  (1976)  showed  how  to  do  this  for  r  <  2,  when  X  is  a  discrete  time 
Markov  chain  (DT.MC)  or  a  continuous  time  Markov  chain  (CTMC)  with  countably  many  states.  Glynn 
and  Iglehart  (1984)  showed  how  to  calculate  such  quantities  when  X  is  an  SMP  with  countably  many  states, 
but  with  the  restriction  that  /*  =  i  =  l,...,m  and  A  =  h, i  =  m -b  1, . . .  r  for  some  g,h  and  m  <  r. 
In  this  report,  we  will  show  how  to  obtain  the  same  quantities  for  the  more  general  cases,  namely,  without 
the  restriction  that  only  two  different  /,'s  can  appear  in  £0^=1  Y(Ii]-  Hy  exploiting  some  combinatorial 
relations,  we  will  prove  three  recursive  moment  formulas,  one  for  SMP,  one  for  DTMC,  and  one  for  CTMC. 

We  now  outline  the  material  to  be  covered  in  the  following  sections.  We  begin  in  Section  2  by  defining 
the  notations.  In  Section  3  we  will  explore  some  combinatorial  relations  and  obtain  a  recursive  formula  of 
£  n[z:i  Y(fi]  for  the  SMP  case.  In  Section  4  and  Section  I,  we  will  prove  two  more  recursive  moment  for¬ 
mulas,  one  for  DTMC,  and  one  for  CTMC.  Finally,  in  Section  6,  we  will  briefly  discuss  some  generalizations. 
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3.  Notations 


To  state  our  result,  denote  Q  =  {Q(z,u,t)  :  z, y  €  £}  to  be  the  usual  semi-Markov  kernel,  P  =  {P„  :  z, y  € 
£)  to  be  the  transition  matrix  of  the  underlying  Markov  chain  R  =  (R^  :  n  >  0)  of  X;  and  let  and 
be  a  matrix  and  a  function,  respectively,  defined  by  (We  follow  the  notation  given  in  Glynn  and  Iglehart 


(1984).) 


fri,*«i.(*.W). 

0, 


if  y  # 
if  y  =  z; 


and 


»€£ 


where 

/!«(*,  y)=/  <“£(z,y,d<), 

Jo 

and  £(z,y,  1)  =  Q(z,  y,  1)/P,y.  For  convenience,  we  assume  that  F(z,y,  0)  =  0,  for  all  z,y  €  E. 

Following  Hordijk,  Iglehart,  and  Schassberger  (1976),  we  consider  vectors  such  as  (t>(0),v(l),.,.,r(i(:)) 
to  be  column  vectors.  In  addition,  for  vectors  u  and  r,  the  symbol  uov  denotes  the  Hadamard  product  of 


vectors 


(«(0)t-(0),«(l)v(l) . u{k)v{k)); 


for  a  matrix  A  =  (oq,  Oi, . . . ,  o„),  set 


u  o  .4  =  A  o  u  =  (u  o  ao,u  0  m, ... ,  u  oom); 


and  for  a  matrix  B  =  (6o,f»i . I>m).  set 


4oB  =  Bo4  =  (aoo6o.Oiofti,...,Om  o6„). 


Finally,  for  vectors  /i,  /j . define 


S'=i/.  =  /i  o/a  0  •  •  •  o  /,, 


and  set  u®(  )  =  (1.1 . 1 )  and  u""*"*  =  u  o  «"  =  «"  o  u  for  n  >  0. 


O 


We  begin  by  noting 


fT(i) 

>'(/)=/ 

yo 


=  r  f(x(t))iiT>,)dt 

Jo 

=  E/  /wo)i{T>o‘^< 

asrO 

00 

=  /(•^)  !{*>•)  (fit+i  ~  (»)i 

«=o 

where  is  the  nth  jump  time  of  the  SMP,  (o  =  0;  ^  is  the  length  of  the  first  r-cycle  for  the  underlying 
discrete  time  Markov  chain  R-,  lx(w)  =  l,w  €  i4:l^(a;)  =  0,w  ^  A.  Since  Y{/)  can  be  written  in  the 
form  as  follows  that  we  can  write  n'=i  ^lA)  =  E“=o  •  Er,=o  n'=i  /•{RnJlSm.+i  - 

To  make  inference  of  the  above  product  form,  we  introduce  the  following  notations.  First  of  all,  let 
N  5  {1,2,3,...},  N,  =  {0}  U  N,  and  H  {(«!,.. ..n,)  :  rij  6  N,,  1  <  »  <  r}  for  each  positive  integer  r. 
Then  for  5i,  5j,. . .  ,5*  to  be  a  partition  of  iV,  =  {n  :  n  <  r, n  6  Af },  define 

[(rti . nr)I  s  Af,', 

[(ni,...,n,)I  =  {(n,,...,n,)  €  iV;  :  n,  =  •  •  •  =  n,}, 

[(n,  :  I  €  Si)  <  •••  <  (n,  :  I  6  5t_i)  <  (n,  ;  i  €  S,)] 

=  {(ni,...,n,)  €  at;  :  n,  =  r»s..»  €  St,  1  <  fc  <  <  -  1; 

n^,  <  •  •  ■  <  n5,_,  <  n,,i  6  S<;  for  some  integers  ns,,  1  <  •  S  f  “  1): 

((n,  :  i  €  Si)  <  •  •  •  <  (n,  :  I  €  Sc_i)  <  (n*  :  i  e  Si)) 

=  {(ni, . . .,  n,)  €  AT'  ;  n,  =  ns^,i  €  St,  I  <  k  <  I  -  1; 

”s,  5  •  ■  •  <  "s«-i  ^  ”«i»  €  St;  for  some  integers  ns,,  1  <  •  <  f  -  1); 
and  for  arbitrary  function  /  ;  — »  R,  and  arbitrary  subset  A  C  define 


for  example, 


/|<»1 . -r)!  =  12  . 

(•i . »»)€l<«t . ■»>1 

/u«l . «r)l  =  y  /(Wl,  .  .  .  ,  rtf  ), 

. . . «r)I 


etc;  and  denote  |/|^  =  XT,. . ,,)€4  l/("« . 

S.  A  recurtlve  formula  for  Mml*Markov  proceaaes 

We  proceed  via  a  series  of  lemmas. 

Lemma  1. 

f  —  I 

M(«i . "01  =  12  52  *|(«-:*€S)<(i.,:i€At»\S)l  +  . "Ol- 

,  =  l  s.  s 

Sin, 

Proof;  To  exploit  the  idea  of  the  lemma,  we  only  prove  a  special  case  that  r  =  3,  while  the  general  case 
follows  easily  by  using  the  same  argument. 

For  r  =  3,  the  lemma  reduces  to 

To  prove  this,  we  have  to  show  that  the  seven  sets  on  RHS  are  non-overlapping,  and  the  union  of  them  equals 
the  set  on  LHS.  The  first  part  is  obvious,  since  the  first  (second,  third,  respectively)  term  on  RHS  means: 
among  the  parameters  {ni,  nj,ns},  ni  (nj,  nj,  respectively)  is  the  unique,  least  element.  While  the  fourth 
(fifth,  sixth,  respectively)  term  on  RHS  means:  among  the  parameters  {ni,n3,n3},  r»i  =  nj  (rii  =  nj, 
nj  =  r«s,  respectively),  and  nj  (nj,  fii,  respectively)  is  the  unique,  greatest  number.  And  the  last  term  on 
RHS  means:  ni  =  nj  =  nj.  Because  these  event  are  mutually  exclusively,  this  proves  the  first  part. 

To  prove  the  second  part,  we  have  prove  that  for  any  (ni,nj,nj)  such  that  0  <  ni,n],n}  <  oo,  it  must 
belong  to  at  least  one  of  the  seven  sets  in  RHS.  This  should  be  obvious,  since  among  any  three  numbers, 
either  there  exists  a  unique,  least  number,  or  two  of  the  numbers  are  equal,  and  they  are  strictly  smaller 
than  the  third  one;  or  otherwise,  the  three  numbers  must  be  equal.  This  completes  the  proof. 
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Corollary  !• 


.=1  Siisi.y 
SQS, 


€S,\5)1 


+  l((«,;i€5,)<  <(«i:i€S,-,)<(«,:i6S.)l- 


Proof:  Exactly  as  Lemma  1. 


Corollary  2.  //|/||(n, . „,)]  =  E(,, . «,)€/v:  IA"i . «f)l  <  »  then 


/(("i . «01  =  H  /((«.:-€S)<(«i:*ew,\5)l  +  /l(. . »,) 

SllSimj 

SQNt 

Proof:  From  Lemma  1  and  the  absolute  summability  assumption. 

Corollary  8.  W  |/)((,...e5,)<  <(»,:.€S,-. )<{«,:.es.)|  <  <»• 


/({«,:ieSi  )<  €S,-  I  )<(«,:i€S,>j 


+  /l(ni:.€5,)<  <(>i.:t€S,- ,  )<(»,:ies.)l- 


Proof:  Using  Corollary  1. 


For  fixed  /,, »  =  I . r,  and  fixed  A  C  .VJ.  we  define  L  as 

. . .  •=«  * 

where  5  is  the  length  of  the  first  i-cycle  for  the  embedded  DTMC  R. 

Umma  2.  . ..)€>:  ^{n'=i[|/.(^.)l(^,+i  -  <» 


El[Y{f,)=L^^^ . 


=  E  E  ^((**:i€5)<(f»,:i€.VAS)l  +  ^((n, . •,)! 


Proof;  The  first  equality  follows  from  the  definition  of  L;  the  second  equality  follows  from  the  absolute 
summability  assumption  and  Corollary  2  of  Lemma  1. 


and  £([l(/i.=j)  l(»>n)]  = 


n=0  ^  1=1  ^ 

= n  ^  /•(»]'*'(->)*{«>»»  i{/i.=j}  I 

11=0  <=i  ^ 


u=0)^E  1=1 

~lS  (3T=|/i  °  ^f)]! 
11=0 

=  [EZ(®UJ.O0r)],. 


The  first  result  follows  immediately. 

Ths  pioof  of  the  second  part  of  this  theorem  is  very  similar  to  Theorem  (3.1)  in  Hordijk,  Iglehart,  and 
Schassberger  (1976),  and  thus  omitted  here. 

Remark;  An  immediate  consequence  < '  this  lemma;  U  I2«o  ~ 

EY{f)  =  i[„]  =  Golf  ®  ^»I  =  EZ(f  0  0i).  This  is  the  first  part  of  Theorem  (5.14)  in  Iglehart  [3]. 

RemArk:  If  E  is  finite,  then  GJ  -•  0  as  n  -►  oo,  thus  the  uniqueness  is  automatically  satisfied,  and  also 
EZ  =  Er=o  G'o=(l-  Go)-\  thus  . =  (1  -  Go)-‘l®:=i/.  » 

Lemma  4.  IfY2{n, . i«,)€l(ni:*€Si)<  <(".:<esi)l  ^^ni=i[l/i(^i)|(f>»i+i  ~  <00  fhen. 


il(n,:ieS,)<  <(n,:i€S,)l 

-X)  CO  oo 

=  E  Go"’(3.6S./.oG|5.|  E  GrM®.€S,/.oG|s,r"  E  GrM®.€5./.0%.|)  -)) 

H5,  =0  "S,  =0  »I,=0 

=£2(  3ces,  /i  o  G|.s,  |EZ(3,g5, /,  »  G|5,|  •  •  •  EZ(S,e5,/i  ®  •••))■ 

Proof:  This  l^mma  can  be  shown  by  using  a  similar  path  decorppocition  argument  as  in  Lemma  3.  We  omit 


the  details. 


Remark:  Lemma  3  is  a  special  case  of  Lemma  4. 


Remark:  If  E  is  finite,  then  iKii.iieSiX  <(«,:t€5,)l  =  (1  ~  ISigSi /•  “^|Si|(l  ~  ® 

Gis,|---{l-Go)-‘(3.es,/.o5|s,|)---)). 

Theorem  1.  For  an  irreducible  SMP,  and  arbitrary  integer  r,  il  53(„, . »,)eJV:  ^{n[=ill/i('^JI(fi«,+i  ~ 

<  00  then, 

(1)  for  each  r'  <  r,  and  EZ(f)  =  ^o/-  1**^* 

r* 

£{[]>■(/.)}  =  . ..-H 

1  =  1 

r'-l 

=  E  E  EZ(&.€sf,oG,E{  n  y{f>)})  +  EZ{®:'^J.o3,.) 

,  =  i  ,eN,.\S 

=  E  ®'es/.oG,£{  n  r(/.)}  +  ®:l,/,o 

\;  =  l  S.IJS  <e.V,»\5  ' 

S  Si 

(2j  for  arbitrary  partition  5i, . . . ,  5((/  >  2)  of  Sr, 

=EZ(  S,gsi  /i  o  G|5,|£Z(0,€s, /<  o  C7|s,|  •  •  •  £Z(®,gSi-,  /•  °  f^|Si-i  |  £(11 

les, 

(S)  for  arbitraiy  nonempty  proper  subset  S  of  Sr, 

^l(n.:ieS)<(fi.:ieVr\S)l  =  ■E2(Si€s/i  oG|si£{  11 

<e^r\S 

Remark:  If  r'  =  1,  (1)  becomes:  EY(/)  =  EZ(f  o  $i)  =  o  ^i)',  when  r'  =  2,  (1)  becomes; 

Fyifi)y(fi'i  =  EZ{fi  o  G\Ey{fi]  +  /j  O  Gi£y'(/|))  +  Ey{fi  o  /;  o  5,).  These  quantities  a^ee  with 
Theorem  (5.14)  in  Igl^'hart  [3]. 

Remark:  If  E  i.s  finit».  in  addition  to  the  existing  assertions,  it  would  also  be  true  if  we  substitute  (1  -Go)~‘ 
f'  r  EZ  in  all  the  three  assertions. 

& 


Proof:  Let  ri  =  r'  in  (1),  rj  =  |5f|  in  (2),  and  rj  =  |JV,  \  S|  in  (3).  We  will  prove  this  theorem  by  induction 
on  fi.rj  and  rj;  and  the  absolute  summability  assumption  will  justify  all  the  following  interchanges  of  £'s, 
/’s,  and  J^'s. 

Induction  basis: 

For  ri  =  1,2,  (1)  is  correct,  (cf.  Iglehart,  theorem  5.H.) 

For  rj  =  1,  (2)  is  correct.  (This  is  a  special  case  of  Lemma  4.) 

For  rj  =  1,  (3)  is  correct.  (This  is  a  special  case  of  (2).) 

Induction  step: 

-Assuming  that  (1)  is  correct  up  to  rj  =  fc  <  r,  (2)  and  (3)  are  correct  up  to  rj  =  rs  =  it  -  1  <  r,  then 

for  arbitrary  set  partition  Si, . . .  ,S(  of  N,  with  |S(|  =  fc  <  r,  by  using  the  induction  hypothesis.  Corollary  1 

of  Lemma  2,  and  Lemma  4  (notice  that  the  absolute  summability  assumption  justifies  the  usage  of  each  of 
them),  we  have; 

<(«.:i€S,-,)««.:i€5,)l 

=  1^  X)  <(».:.€S,-,)<(i..:.€S)<(i..;ie5,\5)l 

;Sl  St  S\9j 
sZSf 

+  <(<i.:.€S.-i)<(<»<:ieSi)l 

fc-l 

=  E  E  MEZ(S,^sf.oG,E{  n  !'(/.)})) +  -4(£Z(®.e5./.o%.|)). 

;  =  1  s.s^,  •eSi\S 

S^Sf 

where  All/)  is  defined  as 

.4(v|  =  EZ{2,^Stf,  0  G'|s,|^2(3i€S»/i  «  <^|s,|  ••  •  •^^(®i£s,_,  A  « G|s,-,H/)  •  •  •))• 

Since  .4  is  a  linear  operator  and  (1)  is  correct  up  to  ri  =  k,  the  last  equation  becomes: 

^1(".  :>€S,)<  <(n.:i€S,-,)<(i.i:ie5,)] 

=.4(r{n  1'(A)}) 

>€S, 

=£Z(;^,gs, A  o f^|s,i^'^(3.€SaA  o<^|s,r”^^{®.es,-,A  ^(A)})---))- 

<€5, 
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This  shows  that  (2)  is  correct  up  to  rj  =  k.  Since  (3)  is  a  special  case  of  (2),  we  also  have:  (3)  is  correct  up 
to  rj  =  k. 


Now,  for  rj  =  A  +  1  <  r, 
k  +  l 

EH  !'(/.)  =  . n,..)l 

i:=l 


=  E 


E 


^l(«<:‘€S)«*,:i€)V»*,\S)l  . “k+Ol 


=  E  E  £2(®.es/.oG,£{  n  ''(/.)}) + 


Notice  that  the  first  two  equalities  follows  from  Lemma  2,  and  the  last  equality  follows  from  (3)  and  Lemma 
3.  We  also  notice  that  the  usage  of  each  of  the  above  properties  is  justified  by  the  absolute  summability 
assumption.  Since  the  RHS  of  the  last  equality  is  exactly  (1)  with  rj  =  Ik  +  1,  we  have  proved  that  (1)  is 
correct  up  to  ri  =  A  +  1.  This  completes  the  induction  step,  and  the  theorem  now  follows. 

Formula  (!'  can  be  simplified  when  X  has  a  special  structure.  Note  that  when  X  is  a  CTMC,  then 
E(i,  V,d<)  =  X(x)ezp(-A(z)()dt,  for  <  >  0,  so  that  =  n!/(A(2))*  =  •»!»(*).  Hence  G,  =  Go  o/i,  = 

0  Go,  and  dn  =  (Pn  o  Pje  =  'Jn-  We  find  that  (1)  can  be  rewritten  as 

r  f  —  1 

^^{n^'(/-)}  =  E  E  ^2(0.^s/,ov,oGoE{  n  !'(/.)})  + £^^(®r=i/.o»Jr) 

1  =  1  ;  =  l  S'  s  t^SrXS 

,T-l 

"^^(E  E  S>esA  Oflj  oGoE{  J|  Y{f,)}  +  ®'-if,0>tr 

S  =  »  ieN,\S 

S-^Nw 

On  the  other  hand,  for  a  DTMC,  Pn{x,y)  =  1  for  each  x,y  6  E,  thus  =  1  and  G,  =  Go  for  each  n, 
so  that  (1)  takes  the  form: 

f  r-l 

^{n^'(/-)}  =  E  E  EZ{®.^sf.oGoE[  n  y{f,)))  +  EZ(Q:^J.) 

1  =  1  j  =  l  S.  s  >^S,\S 

S  s 


=  EZ 


E  s.es/.oGoE{  n  r(/.)} +  ©:=,/, 

S'  S  -j  ie.v,\s 

S 


0 


Notice  that  when  r  <  2,  the  above  two  special  cases  reduce  to  the  formulas  given  in  Hordijk.  Iglehart,  and 
Schassberger  (1976). 

The  above  formulas  are  based  on  the  special  structure  of  X.  Dther  reductions  are  possible,  for  example, 
if  /,  =  /,  for  I  =  1, . . . ,  r.  then 


E(Y[J)Y  =  '^(^\EZ(r  oG,E(Y(J)y-^)-^EZ(r 

j=i 

=  ^g,e{y{!)y-’  + 


On  the  other  hand,  if  /,  =  y.  i  =  1 _ _  m; /,  =  fc,  i  =  m  +  1, . . . ,  m  +  n,  then 


E{Y{g)r(Y[h)Y 

=  X!  h)('')EZ{g'ch^oG,+,E{Y{g)r-'(Y[h))*-^)  +  EZ{o”'oh'o^„^,) 

0<i<  «  \  *  /  / 

OKi* 

jS  ■ 

C<t*  j^m^m 


Notice  that  this  is  exactly  the  equation  (2.3)  in  Glynn  and  Iglehart  (1984). 


4.  A  recursive  formula  for  discrete  time  Markov  chains 

Let  {A'fc  :  >  0}  be  an  irreducible  DTMC  with  countable  state  space  E,  and  one  step  transition  matrix 

P  =  {P,y  :  i,y  €  E).  For  fixed  z  €  E,  let  T{z)  —  inf{n  :  z,X^  =  z}  and  set  (notice  that  we  will 

always  suppress  the  z  in  r(z)  when  there  is  no  confusion) 


T-l 

rU)  =  E 

fcssO 

where  /  :  £  — •  iZ  is  an  arbitrary  function.  We  wish  to  study  mixed  moments  of  the  form 

r 

«sl 

when  /,:£—•/?  is  an  arbitrary  function  for  i  ^  and  r  is  a  positive  integer.  We  notice  that 

>'(/)  =  Er=o/(-v*)  =  Er=o/(  Afc)l{7->/t}.  this  is  almost  of  the  same  form  as  the  decomposition  for  Y(f) 
given  before,  this  gives  us  the  motivation  to  follow  the  development  in  the  SMP  case  quite  closely.  We  also 
need  the  following  modifications  of  notations: 


if  y  5^ 
if  y  s  z. 


And  revise  L  into  (D  for  DTMC)  such  as: 


. n,)eA  •=! 


(r>», 


for  each  arbitrary  set  A  C  iV'.  Notice  that  an  immediate  consequence  of  this  definition  is;  l^(/i)  = 

Lcnunc  5. 


1|<" 


r-t 


*’)\  =  E 

J  =  l 


(-1)^  +  (-1)'  . n,))- 

5il5l>^ 

JVf 


[(n,,.,..n,)]  =  |J(n,  <  {n, 

»=l 


W  6 -V,\  {,•})], 


Proof:  We  first  notice  that 


since  at  least  one  of  the  element  in  parameters  must  be  less  than  or  equal  to  the  rest;  and  each 

set  of  RHS  is  a  subset  of  the  set  in  LHS.  Define  Wi  =  (n,  <  (ny  :  j  €  Nt\  {•})],  then 


M(». . «0I 


,  =  1  5ilSI=> 
^  SQS. 


=E  E  (~l)^“‘l[(»i:*€S)<<«<:<€JV,\S)l  +  . «') 


1  =  1  5ilS|*j 


This  completes  the  proof. 


We  now  state  a  sequence  of  corollarys,  lemmas,  and  theorem,  which  can  be  regarded  as  the  “<"  version 
of  their  counterparts  in  Section  2.  We  will  omit  all  the  proofs  because  of  the  similarity  between  their  proofs 
and  their  counterparts’. 


Corollary  1. 


€S,\S)1 


*  — 1  S!l5i«y 


+  <(«,:.€5, -,)<(«, :ieS.)l- 


Corollary  3.  I/|/l[(,, . =  E(, . l/("i . "f)l  <  » 


. o.)!  =  13  13  . •>) 


}  =  l  S'  s 

S^Nr 


13 


Corollary  S.  l/li(«.:iesi)<  <(i*.:<6s»-i)<(»i:«€s,>i  <  <». 


lS.1-1 

j  =  I  SilS  -J 
SS5, 


+ 


i:<€Si)<  <(*i:*€S, -!)<(•, :i€Si)l- 


Lemma  8.  . «,)€^:  ^^{n'=ill/.{-Y«.)ll{T>«,}l}  <  » 


EflY(/.)  =  q, . Ml 


f 

"E  E  (-1)^  ‘^U»i:*€S)<(«(:i€^AS)l  . ».)1- 

j~i  SilSI-J 


Corollary  1.  If 


E  r{ni|/.(X..)|l(r>.al}< 

(oi . S(<ii:t€5.-,)<(«i:i€S)<(»i:<€S.\S>l  •=! 


then 


^1(".:<€S,)<  <(<«i:i€S,-i)<(«i;i€5,)l 


|5,|-l 

=  E  E 

j=i  j.if-j 


<(»,:i€S,-,)<(»(:i€S)<(»i:.eSi\S>l 


+ 


O 


Lemma  7.  // ^^“=0  ^^(10:=,  l/.(A-,  )|ll,r>n( }  <  »  then 


■''U-i, . »,)i 


=  E  f^(3:=i/.i  =  =  -5^(0:=,/.). 

•  =0 


Mojrovfr,  the  vector  u  the  Qoiqae  solotioa,  y,  of 

V  —  0^=1 /<  +  ^oV 
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satisfyiay  G"  1/  -►  0  as  n  -»  oo. 


AemarJc;  For  r  =  1,  this  lemma  says:  EY(f)  —  71T=o^f  ”  ^^(/)-  Thus  for  DTMC,  if  G5(-)  is  absolute 
summable,  then  EY{-)  =EZ{-). 

Remark:  If  E  is  finite,  then  GJ  -»  0  as  n  -•  oo,  thus  the  uniqueness  is  automatically  satisfied,  and  then 

. „,),  =  (i-Go)-‘i®:=i/.i- 

Lemma  8.  ff  . ii,)€l("i:*€Si)<  <(«,:ies.)l ^^nt=ill/*(-^»()|l{s>«i}l}  < 

^IC".;>€S,)<  <(»,:i€S.)I 

=  *^0*‘ (®*€S,/t  ®  <^0*^’(®t€S,/t  »•••»  53  ^0*^'(®'€S, /•)•••)) 

«s,=0  lls,=0  «s,=0 

=EZ  ^®i€Si  li  o  EZ{®i^Stfi  o  £Z(®igs,/i)  ••  0^ 

=EY  fi  0  EY{®.es,f,  o  •  •  •  o  EY{®i^s.f,)  •••))• 

Theorem  i.  For  a  irreducible  DTMC,  aadarbitrairiateferr,  . »r)€N;  ^{ni=ill/»(-’^«()U{T>i*,)l} 

<  00  then, 

(1)  for  each  r'  <  r,  and  EZ(f)  =  ZT=o 

f* 

EdlW.D^i,^. . 


1=1 


r'-l 


.€.v,,\5 


J  =  l  St'S  -J 

ss.v,, 


=  EZ 


(E  E  (-lH-‘s.es/.oE{  n  J'(/.)}  +  (-ir'~‘ ©:=./.): 

';  =  l  Sll5  ./  leJV.AS  ' 

SiJW,,  ' 


(2}  for  arbitrary  partition  S|,...,S((<  >  2)  o/.V,, 

^KtoeSil^  <(>«(:<€S,-,)<{«i:.€S.)I 

=£:Z(3,€s. /.  o  £Z(S.€s,/.  o  •  •  •  0  £^^(S.€5... /.  o  £{  n  >’(/.)})  ■ '  )); 

•€Si 
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(3)  for  arbitrary  DODcmpty  proper  subset  S  of  Nr, 


®  JJ  !'(/•)})• 

•€/V\S 


Remark:  Since  EZ(-)  =  EY(-)  if  any  of  them  converges  absolutely,  under  the  condition  of  this  theorem, 
each  assertion  is  correct  if  we  substitute  EY  into  EZ. 

Formula  (1)  can  be  simplified  when  fi's  have  some  special  structure,  for  example,  if  ft  =  /,  for  •  = 
then 

E(y(f]Y  =2  (')(-ir‘£z(r  0  £(i'(/)r-^) + {-ly-^Ez^n 

='^^(2  oE(r(/)r> +(-ir‘r). 

'j=t  ' 

Notice  that  this  is  exactly  the  equation  (2.6)  in  Glynn  and  Iglehart  (1984).  On  the  other  hand,  if  /,  =  j;,  i  = 
1 . m;  /,  =s  fi,  I  =  m  +  1, . . . ,m  +  fi,  then 

i:(i"(a))"’(i'(/i))" 

=  E  ('”)('’)(-i)"^'"‘£^{ff’oft^o£(nii)r"’(i'(fc))"*'’)  +  (-i)’"'*’"”‘^^^(a"’'>'»’*) 

9<*<m  \  •  /  \^/ 


9<J<» 

9<i*j<m4 


E  oE(Y(g)r-'(Y{h))*->  +(-ir+’'-^r 


5.  A  recunive  formula  for  continuous  time  Markorv  chains 

We  will  show  how  to  get  another  recursive  moment  formula  for  CTMC  in  this  section.  First,  let  A'  = 
{A(l)  ;  1  >  0}  be  a  CTMC  with  countable  state  space  E,  transition  matrix  F(<)  =  y  6  E},  and 

Q-matrix  Q  =  (qjy  ;  r.y  C  E}  as  the  infinitesimal  transition  parameters.  Recall  that  in  a  continuous  time 
case  Q  =  P'(0)  is  the  given  data  of  the  model  and  that  P(<)  is  generally  hard  to  calculate  and  rarely  given 
explicitly.  The  exponential  distribution  holding  time  in  any  state  x  €  E  has  mean  ^7*  =  all  x  e  E, 

we  assume  that  0  <  7,  <  oo,  so  that  all  states  are  stable  and  nonabsorbing.  In  addition,  we  assume  that 
=  0,  which  guarantees  that,  starting  from  any  state  x  e  E,  the  CTMC  makes  a  transition  to  a 
next  state  y  €  E.  The  element  of  the  jump  matrix  G  =  {G(x,  y)  :  x,  y  e  E }  of  A  are  defined  by 


We  will  assume  that  G  is  irreducible.  Notice  that  this  is  equivalent  to  A  is  irreducible,  and  therefore,  positive 
recurrent.  For  fixed  z  €  E  as  the  regenerative  state,  let  T(z)  =  inf{/  >  0  :  A(l-)  ^  A(l)  =  z}  and 


!'(/)= 

Jo 


where  /  :  E  -*  P  is  an  arbitrary  function.  (Hereafter,  we  will  suppress  the  z  in  T[z)  when  no  confusion  is 
likely.)  Our  objective  is  to  study  the  mixed  moments  of  the  form 

El[Y(f.), 

•=i 

when  /,  :  E  —  P  is  an  arbitrary  function,  for  1  =  1, 2, . . . ,  r,  and  r  is  a  positive  integer. 


We  need  to  define,  as  in  Iglehart  [3],  oPi»(0  =  Pi{P  >  f.A(<)  =  y),  and  oP(0  =  {oPi»(0  :  x,y  6  E}; 


also  let 


\0,  ify  =  z. 


.\nd  let  (T  =  . (Tr)  to  be  a  permutation  of  (1,2 . r);  _ (fjr},  where 

{<r,2 . (Tjr)  is  a  permutation  of  (!,...,>  -  1,^  +  1 . r);  and  =  (ff,!, . ..  to  be  a  permutation 

of  (1 . -  1,>+  1 . r). 

From  the  above  definition,  we  immediately  have 

{<^}= 


which  means;  the  set  of  all  permutations  of  ( 1, ....  r)  is  the  union  (over  all  >’3)  of  the  sets  of  all  permutations 
beginning  in  j.  As  a  consequence 

R+  =  :  0  <  »,  <  00, i  =  1 . r) 

=  . ^  ^  ^  ^ 

a 

t 

=  U  . •t'r)  e  /?;  :  ry  <  <  •  •  •  <  }. 

J=1  Oi 

We  also  notice  that  the  sets  in  the  RHS  are  almost  mutually  exclusive,  in  the  sence  that  each  pair  of  sets 
have  an  intersection  which  is  Lebesgue  measure  0. 

We  begin  by  citing  a  theorem  from  Hordijk,  Iglehart.  and  Schassberger  (1976). 

Lemnui  9.  (Hordijk,  IglebArt,  And  Scbissberger  (1976).) 

EYif)  =  r  oP{t)fdt  =  f;GJ{/0,-‘)  =  EZ(foq-^). 

provided  that  the  integral  (or  eqaivaJently,  tbe  sam)  converges  absolutely. 


Lemma  10.  l/<».{*^(“’<r.))|l{r>«-,,) }  <^“^<7,  •  •  dwg,  <  00,  (hen 

,  in/-  .(A'(ttvJ)l(7->„,  dwg, 

•'*'»!  S  Sv*.  ,=  | 

=  51  o«"‘  o  0  j-‘  0  -  -  o  ^  og-‘ 

»i=0  fip=0 

=  EZ(f„,  oq-'  o£Z(/„,  og-'  o  -  oEZ{f„,  oq-').-.)) 
=£¥{/„,  oEY[U,o..-oEY(fa,)---)), 
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Proof: 


■L 


lv,^=0  Jw,,>v,^  ^*>•,>*‘•,-1  t=i 


El  f  {fl  /<T.(-Y{«r,,))l(T>«,,)}‘^»<f. 

/»,,<  <m/„  ,_J 

=  /  ^‘{fl  (A'(ttv.))l{r>«,,)}‘^*'<r,  ■■■dw„ 

<«'•.  1=1 

-r } 

^  «/«l  =0  JWt 

^oo  ^oo  ^oo 

-I  I  ‘"I  Et{  ^  /,,  (ii)l{r>«>,,}l{X(  •»•,  )=ji) 

^«),,=0  Vi,,,=0  Jw,,=0 

/ffi(A)^^{7'>«i,,+m»,,)  l{X(w,,+«i,,)=ij)  ■■■ 

/i€£ 

/»f(-^''')l{r>w,,+  +«<»,}l{x(w»,+  +«r,,)=i,)}‘^“Vi  ••• 

u€E 

/OO  i>(Xl  /OO 

I  ■■■  /  /<f|  (/>)o^lll  (“’»!  )  ) 

;,,=0/w,,=0  •'«'»,=0j,€£  „g£ 

■••  /»r(ir)oPy,-.;,(»<T,)dw<r,  •  •  ’  dw,, 

J.€E 

/oo  ^oo 

53  /<T,(il)oPlj.(«^9.)<^«iV,  /  51  /»»(>*)oP;i 

■■■  /  ^3  f<^r(j')oP},-l}ri^'o,)  d^’(r, 

•  —0  .  ^  F 


=  5Z  E  or' »•••'’  53  <^'/.„o,-'], . 

fia=0  «v=0 


>ii=0 


where  our  absolute  integrability  assumption  justifies  the  various  interchanges  of  £’s,  /'s,  and  J^'s;  and 
Lemma  9  is  used  r  times  to  obtain  the  last  equality. 

This  completes  the  proof. 
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Theorems.  I//^.  £{0'=!  <00,  then 

£nr(/.)  =  £^  /  - 

t=l  a  •'*’•1$  I**,#  ,=1 


r*  CO 

1=1 11=0 

00  f' 

«=o  i=l 

f* 

=  £z(E/-  05  *  o  £  nw,)] 

1  =  1 

T* 

=  EY[Y,f,oEl[Y(f,)], 

1  =  1  jXi 


I  for  any  r'  <  r. 

I 


Remark:  When  r  =  2,  the  theorem  reduces  to 


00 

£>'(/i)i'(/3)=  (A  o,-'o£r(/,)  + /,o,-'  o  £!'(/, )]. 

i»=0 


This  agrees  with  equation  (3.13)  in  Hordijk,  Iglehart,  and  Schassberger  (1976). 


Proof:  We  prove  this  theorem  by  induction. 


Induction  bati$: 


From  Hordijk,  Iglehart,  and  Schassberger  (1976),  the  assertion  is  correct  for  r*  =  1. 


Induction  gtfp: 


Suppose  the  assertion  is  correct  up  to  r'  =  A  —  1  <  r,  consider 


El[y[f.) 

1  =  1 

-E  f  /i(-V(tt’i))l(r>«,i 

JR^ 

Jr. 

r  * 

1=1 

,  =  l  <r,  J}<^-„<  .  =  1 

k  .  * 

J  =  l  IT,  1  =  1 


*  .  rc 


=  ZL[  H  <^‘A09‘‘o  X! 

;  =  1  «J=0  Biso  »f=0 

=  E  £  ^‘/;or‘°[E  £  c;v.„or‘o  -  o  £  g;*/.,,.  <>»-*] 


;=1 «1=0 


|*J=0 


=£[£^/^<’«'‘‘’'^n^(/')] 


j  =  l  n=0 


oo  _  k 


n=0  J  =  l 


where  the  absolute  integrability  assumption  justifies  the  v^ous  interchanges  of  /'s,  ajid  E's\  Lemma 
10  justifies  the  fifth  equality;  and  the  last  equation  is  obtained  by  the  induction  hypothesis. 

This  completes  the  induction  step,  and  the  theorem  follows. 
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0.  Generalliatlon  and  dlscuMlon 


Wp  will  discuss  some  generalizations  in  this  section.  As  before,  let  X  =  {A'(f)  :  <  >  0}  be  an  irreducible  semi- 
Markov  process  (SMP)  on  countable  state  space  E,  and  R  =  {iZ,  ;  n  >  0}  be  the  underlying  Markov  chain  of 
A'.  For  each  f,  denote  X'(t]  m  X(t  +  r),  where  f  =inf{f'  >  0  :  X{t  +  t')  A(f)};  namely,  A’(0  is  defined 

as  the  "next  state’  process  of  X  at  time  1.  For  fixed  z  €.  E,  let  T{z]  =  inf{/  >  0  ;  A(t  — )  ^  z,  A(t)  =  z), 
i5(z)  =  inf (n  >  0  :  Rn-i  ^  z,R„  =  z}  and  set 

y'(n=  r^^\'{x{i).x‘(t))di, 

Jo 

where  /'  :  £  x  £  — »  i?  is  an  arbitrary  function.  (Hereafter,  we  will  suppress  the  z  in  r(z)  and  fi(z)  when 
no  confusion  is  likely.)  Our  first  objective  is  to  study  the  mixed  moments  of  the  form  '^hen 

f[  :  E  X  E  R  are  arbitrary  functions,  for  i  =  1,2,  ...,r,  and  r  is  a  positive  integer. 

We  begin  by  noting 

}"(/')=  f'ix(t),x'{t))dt 
Jo 

=  r  f'(X{t),X•(t))\^r>t)dt 
Jo 

=  E/  r{X(tlX'{t)}Ur>ndt 

»=o 

'X> 

f»=0 


it  follows  that  n'=i  y'if.)  =  Er,=o  Er,=o-  'Er.=o  nUi  -  f«,)l(4>«,)-  Next,  we 

define  L'  as  follows;  for  fixed  ,  i  =  1, ....  r.  and  fixed  A  C  .V',  let  L'  be 


'(•I . •,)€■*  •  =  ! 


where  t  is  the  length  of  the  first  z-cycle  for  the  embedded  DTMC  R. 

We  proceed  via  a  series  of  lemmas.  .Vote  that  the  proofs  for  them  are  very  similax  to  the  proofs  for  their 
counterparts  in  .Section  3  and  thus  omitted  here. 


Lenun*  11.  . „,)€.v;  ^{n'=,[|/.'(^.-^.fi)|(f.^+i  “  <  <»  tAen 


Eli  >"!/.')  =  ill . . 


1=1 


€5)<(«<:*eWr\S)l  +^[(«i . «,))• 


j=t  s.is'-y 
SCN, 


Corollary  1.  If 


then 


»,+l  -  fl•.)l^^>ll,}l}  <  * 

("1 . )<  <(«.:i€Si-i  )<('».:'€Si)l  i=l 


f  < 

^((•ilceSiX  <(«.:i€S,-,)<(«,:.€5,)l 


IS,  I- I 


-  Yi  Yi 

s;s, 


<€S,)<  <(«.:,eS,-,)<(«.:,eS)<(i»,:t€S,\S)| 


+  ^[(ncieSiX  <(<«j:i€Si-iX('»,;'€S.)|- 

Umma  13.  // E«o  /?n+i  )II{^,+i  -  C.ri{»>.}}  <  oo.  and  define  EZ(g)  = 

then 

■Xl 

ii(n. . „)l  =  E  ^  °  t*r  o  P)e]  =  EZ[{q:^JI  o  o  P)e]. 

11=0 

Aforeover.  fhe  vector  ^  unique  solatioa,  y,  of 


V  =  °t*rO  P)e  +  GoV 


satisfyiag  G^y  —  0  as  n  —  oc. 


Bemark:  A  consequence  of  this  lemma;  If  ErLo^{|/*(-^'^+i)|{  f«+i  -  }  <  oc,  then  £!"(/')  = 

if,!  =  E:Lo  ^K/' °  PH  =EZl(/'o  ft,  oP)e]. 


Remark:  We  notice  that  by  setting  /'(i,  y)  =  f{x)  for  all  y  €  £,  then  (/'  o  y,  o  P)e  =  /  o  di-  Thus  we 
immediately  have  another  consequence  of  this  lemma;  If  ^{l/(^)|((ii+i  ~  then 

pyif)  =  Er=o  G2[fo$,)  =  EZ(fo0,).  This  is  the  first  part  of  Theorem  (5.14)  in  Iglehart  [3], 


.  W-  ;  TT-  ;  ^ 


I 


EemArk:  If  E  is  finite,  then  GJ  -►  0  as  n  -•  oo,  thus  the  uniqueness  is  automatically  satisfied,  and  also 
=  thus  ij(., . ,,),  =  (l-Go)-‘l(®:=,/;o/i,oP)e]. 

Lemma  IS.  . i»,)€l(".:>€Si x  <(»,:i€5i)|  •^{n«=iIl/«  (^i< ^  ^  tien, 


I 


I 


^l(".:<e5,)<  <(n,:.eS,)l 

=  E  [is.es. /:oG,5.|)  E  G:">[(3.€s,/:oG,s,|)---  E  ^^r'l(®.€S./>/i|s.|0PH---]] 

t«s.=0  tis,=0  »s,=0 

=£Z  [(s.es,/.'  °  G|s.|)£.^((®.esi/,'  o  G|s,|)  -  ® /'isii  “  ^)«l  •■•]]• 

Remark:  Lemma  12  is  a  special  case  of  Lemma  13. 

Remark:  U  E  is  finite,  then  ■i[,,,:.es,)<  <(«,:.6S.)|  ~  “  Go)“*[(©,cs,/,' o  G|s,|)(l  -  Go)"‘[(®,es,/,' o 

G|s,i ;  •  •  •  ( 1  -  Go ) - ‘  1(  ®.es. /;  o  /i|s. I  o  P)el  •  •  -1] ■ 

Theorem  4.  For  an  irreducible  S\£P,  and  arbitrary  integer  r,  if 

(n . «.)€.v:  .=1 

then, 

(Ij  tor  each  r'  <  r,  and  EZ{g)  =  G"?.  •'«  hare 

. ...)i 

1  =  1 

=  E  E  p2[(3.€s/;»G,)r{  u  r(/:)}]+£:z[(®::,/:oMr.oF)ej 


;  =  l  St  S  mj 
S^S^, 


=  -^^(E  L  (S.€5/.'°G,)P{  n  »"(/.')) +  (S.=./r«>/ie.oP)e); 

S=. 


<€.v,AS 


{2}  for  arbitrary  partition  Si, . . . ,  S((<  >  2)  of  .V,, 


^i(*.:>€S.)<  <(it,:ieSi-i)<<>«.:ieSi)l 


=  £z[(3.es./.'oG|s.|)FZ[(3.es,/.'oG,s,|)---FZ[(3.es.../:oG,s._.|)£{n  n/T)}!---]]; 


•€S. 
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(3)  for  Arbitrary  aoDempty  proper  subaet  S  o/Avi 


€5, €.VA5)1=^^2[(S.€s/.'0G|5i)£:{  H/;)}]. 

i6JV,\S 

Remuk:  If  r'  =  1,  (1)  becomes;  £}"(/')  ■—  £Z{(/'ofi,  oP)e]  =  5Z^G5l(/*  agrees  with 

Lemma  12. 

Remark;  If  £  is  finite,  in  addition  to  the  existing  assertions,  it  would  also  be  true  if  we  substitute  (1  -Go)~‘ 
for  £Z  in  all  the  three  assertions. 

Corollary  1.  For  an  irreduc.bte  SMP,  and  arbitrary  integer  r,  if 

{•i . «.)€>:  <=» 


then  for  each  r'  <  r,  we  bare 


E  I]  r(/,)}]-bFZ [©'I, 

.=i  j=i  s.  s  .,  >e/vAs 

=  Ez(f^  E  (®i€sfioG,)E{  n  >'(/.)}  + ®:Li/.o/>r'); 

*€N,>\S  ' 


Remark:  This  is  exatly  Theorem  1. 


Corollary  2.  For  an  irreducible  S\IP,  and  arbitrary  integer  r,  if 


. . "Oe.v;  •=! 


where  y"'(c)  =  5Z»=o  <'(F»,  R„  +  i ),  and  e,c,  :  E  x  F  — ►  R,  are  arbitrary  Aintions  for  i  =  1,. . .  ,r,  then  for 
each  r'  <  r,  we  have 


=  E  E  ^:2[(-.€sc.pCo)E{  n  F"(e.)}] +Ez[(©:l,e.oP)e] 


;  =  l  J.  5  -j 
5  .  V., 


>e.v,.\s 


=  E  ('^>€sr.oGo)£{  n  r"(e, )}  +  (©:!, e.oP)e); 

'j  =  i  'ey  AS  ‘ 


A  further  generalization  of  Theorem  4  is  possible.  Let  F,  :  £  x  £  -►  i?  be  a  random  matrix  for  each 


I  =  1 . r,  and  let  £[£,]  =  F,,  £[F,  o  F,1  =  F,  o  F,,  £(0iF,]  =  ®,F,,  etc.  By  following  the  same  proedure 

as  in  Theorem  4,  we  will  have: 

Corollary  3.  For  an  irreducible  SMP,  and  arbitrary  integer  r,  if  F,  'a  are  independent  ot  tbe  bolding  time 
{,+1  —  for  each  r»,  and 

r 

^  £{nii^'.(^..^ 

(•I . •=» 

then, 

£(0  nr.))  =  I',,. . ..,1 

1=1 

=  2  E  Ezl(®:^oGy)E{  n 

,=1  .esAs 

/<•->  _ 

=  ^^(E  E  (®.e5F.oG,)£{  n 

\=l  5,IS.,>  i€NAS 

Finally,  denote  Ti  aTj  =  min(Ti,Tj),  then  for  arbitrary  zi.zj  €  E,  E  f'(X(t),X*(t))dt 

can  be  computed  in  exatly  the  same  way  as  in  Theorem  4  by  redefining  G,,  for  each  n,  as: 

r  -  f  ify^iiorz,; 

^"^*’*""10.  ify=z,  orz,. 

Further  generalization  along  this  idea  is  obvious. 
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